This invention relates to a frame processor for use in an encoder pipeline arranged to encode a stream of video frames.
During the real-time encoding of streaming media such as video and audio, it is often useful to be able to modify the configuration of the media encoder. This allows the encoder to adapt to changes in the processing resources available to the encoder and, in the case where the media is streamed over a network, the network conditions encountered by the stream between the transmitter and receiver. For example, in the case of video telephony over a network, a video stream can suffer from transient drops in network bandwidth which lead to buffer bloat problems in routers causing excessive buffering delays and/or packet loss if the encoder does not respond by dropping the bitrate at which the encoder generates the encoded media stream transmitted to the receiving device.
Conventionally, configuration changes or switches at a video encoder are achieved by flushing the encoder and immediately reconfiguring the encoder in accordance with a new encode configuration. However, such configuration switches are not seamless and result in disruption to the encoded video stream. Alternatively, a video encoder can be forced to immediately change its configuration without first flushing the encoder, but this leads to glitches in the encoded video stream, which would typically be observed in the decoded video stream as freezes and frame artefacts. This is because each video frame of a stream takes a finite amount of time to traverse the encoder. When the configuration switch is forced, one or more frames are likely to have been already partially processed under the old configuration; completing encoding of those frames according to a new configuration leads to glitches in the stream.
These issues are not limited to video encoders, with other types of media encoder facing similar problems.
Certain encoders also suffer from unacceptable latencies when performing dynamic switching of frame size. This can be due to the encoder being a software encoder supported at a hardware platform which itself is not optimised for performing frame size switches, or which is accessed by the software encoder by means of abstractions which are not optimised for performing frame size switches. For example, on the Android mobile platform, OpenMAX abstractions are used to interface a software encoder to the underlying hardware functionalities employed to perform aspects of media encoding. The use of OpenMAX abstractions leads to multiple state machine transitions when a dynamic switch of frame size is performed and, as result, excessive latency.